Capture and processing of 3D microscopic images through multi-perspective technology

En las últimas décadas, ha habido un desarrollo notable de nuevas tecnologías de captura de imágenes 3D. Entre estas tecnologías existe una que merece una atención especial, por su capacidad de capturar la información 3D de escenas iluminadas incoherentemente. Esta técnica es conocida bajo varios nombres, según el área de conocimiento en que se investiga: Imagen Integral, Imagen Plenóptica y como es más conocida en ámbito internacional Lightfield Imaging. Desde su invención a principios del siglo XX, y más fundamentalmente desde su renacimiento en los años 90, esta tecnología a captado una atención creciente en la comunidad científica debido a su capacidad para reconstruir escenas 3D a partir de una sola captura. Esta capacidad resulta de su habilidad para capturar en una sola toma, no sólo la información espacial de los rayos emitidos por los puntos que com- ponen la escena, sino también la información angular. Esto, abre el camino hacia nuevos posibles escenarios de investigación y desarrollo no solo del sistema óptico, sino que también de todo el apartado de procesamiento de la información adquirida con algoritmos de reconstrucción más avanzados. Destaca también el interés en muchas aplicaciones que al día de hoy varían desde cámaras de telefonía móvil, para obtener mapas de distancia que permiten, entre otros efectos, el denominado efecto bokeh, hasta la producción cinematográfica de películas. Más recientemente se ha demostrado que los sistemas de imagen integral están en condiciones de proporcionar resultados muy interesantes también en el campo de la microscopía óptica. Sin embargo, esta aplicación todavía presenta limitaciones técnicas in- herentes a la misma naturaleza de la tecnología de imagen integral. En esta tesis analizaremos los principios teóricos de sistemas de imagen integral para microscopía, con particular atención a sus limi- taciones, con el objetivo de promover soluciones que puedan mejorar esos aspectos. Por esto, el enfoque principal de la tesis ha sido avanzar en la tecnología de microscopía de imagen integral en dos aspectos: en el desarrollo ́optimo del sistema ́optico de captura y en desarrollo de nuevos algoritmos de reconstrucción de la imagen 3D.During the last decade new technologies for the acquisition of 3D images have shown an impressive growth. One of these techniques that is worth mentioning, due to its capability of capturing the 3D information in a single shot, is known under different names such as Integral Imaging, Plenoptic Imaging and Lightfield Imaging. Since their invention at the beginning of the 20th Century but mainly after their rebirth in the ‘90s, lightfield imaging systems have gathered the attention of a vast community of researchers thanks to their promis- ing capabilities of capturing 3D structure of incoherently illuminated scenes with just a single shot. These results are achievable thanks to the capability of these systems to capture not only the spatial in- formation of the light rays emitted by the scene, but also its angular information. This has opened new research paths towards the design of improved systems, new dedicated algorithms, and a great amount of new applications, that can vary from phone cameras for bokeh ef- fect, till cinema production for after effects. Lately, Integral-Imaging systems have shown very promising capabilities of capturing the 3D structure of microscopic samples. Nevertheless, there are some tech- nical limitations inherent to this technology that needs to be taken into account. In this Thesis we will analyse the theoretical principles of light- field microscopy with particular focus to its bottleneck limitations with the scope of implementing new design solutions in order to over- come those problems. The aim of this work is to provide an optimal design for 3D-integral microscopy with extended depth of field and enhanced lateral resolution. The principal focus of this Thesis has been to contribute making a step forward to the lightfield microscopy technique, in both directions: optical optimization of the capturing system and development of new algorithms for the reconstruction of the 3D sample

Area-based depth estimation for monochromatic feature-sparse orthographic capture

With the rapid development of light field technology, depth estimation has been highlighted as one of the critical problems in the field, and a number of approaches have been proposed to extract the depth of the scene. However, depth estimation by stereo matching becomes difficult and unreliable when the captured images lack both color and feature information. In this paper, we propose a scheme that extracts robust depth from monochromatic, feature-sparse scenes recorded in orthographic sub-aperture images. Unlike approaches which rely on the rich color and texture information across the sub-aperture views, our approach is based on depth from focus techniques. First, we superimpose shifted sub-aperture images on top of an arbitrarily chosen central image. To focus on different depths, the shift amount is varied based on the micro-lens array properties. Next, an area-based depth estimation approach is applied to find the best match among the focal stack and generate the dense depth map. This process is repeated for each sub-aperture image. Finally, occlusions are handled by merging depth maps generated from different central images followed by a voting process. Results show that the proposed scheme is more suitable than conventional depth estimation approaches in the context of orthographic captures that have insufficient color and feature information, such as microscopic fluorescence imaging

Fourier-domain lightfield microscopy: a new paradigm in 3D microscopy

Recently, integral (also known as lightfield or plenoptic) imaging concept has been applied successfully to microscopy. The main advantage of lightfield microscopy when compared with conventional 3D imaging techniques is that it offers the possibility of capturing the 3D information of the sample after a single shot. However, integral microscopy is now facing many challenges, like improving the resolution and depth of field of the reconstructed specimens or the development and optimization of specially-adapted reconstruction algorithms. This contribution is devoted to review a new paradigm in lightfield microscopy, namely, the Fourier-domain integral microscope (FiMic), that improves the capabilities of the technique, and to present recent advances and applications of this new architecture

The Lightfield microscope eyepiece

Lightfield microscopy has raised growing interest in the last few years. Its ability to get three-dimensional information about the sample in a single shot makes it suitable for many applications in which time resolution is fundamental. In this paper we present a novel device, which is capable of converting any conventional microscope into a lightfield microscope. Based on the Fourier integral microscope concept, we designed the lightfield microscope eyepiece. This is coupled to the eyepiece port, to let the user exploit all the host microscope's components (objective turret, illumination systems, translation stage, etc.) and get a 3D reconstruction of the sample. After the optical design, a proof-of-concept device was built with off-the-shelf optomechanical components. Here, its optical performances are demonstrated, which show good matching with the theoretical ones. Then, the pictures of different samples taken with the lightfield eyepiece are shown, along with the corresponding reconstructions. We demonstrated the functioning of the lightfield eyepiece and lay the foundation for the development of a commercial device that works with any microscope

Optical sectioning microscopy through single-shot Lightfield protocol

Optical sectioning microscopy is usually performed by means of a scanning, multi-shot procedure in combination with non-uniform illumination. In this paper, we change the paradigm and report a method that is based in the light field concept, and that provides optical sectioning for 3D microscopy images after a single-shot capture. To do this we fi rst capture multiple orthographic perspectives of the sample by means of Fourier-domain integral microscopy (FiMic). The second stage of our protocol is the application of a novel refocusing algorithm that is able to produce optical sectioning in real time, and with no resolution worsening, in the case of sparse f luorescent samples.We provide the theoretical derivation of the algorithm, and demonstrate its utility by applying it to simulations and to experimental data

What about computational super-resolution in fluorescence Fourier light field microscopy?

Recently, Fourier light field microscopy was proposed to overcome the limitations in conventional light field microscopy by placing a micro-lens array at the aperture stop of the microscope objective instead of the image plane. In this way, a collection of orthographic views from different perspectives are directly captured. When inspecting fluorescent samples, the sensitivity and noise of the sensors are a major concern and large sensor pixels are required to cope with low-light conditions, which implies under-sampling issues. In this context, we analyze the sampling patterns in Fourier light field microscopy to understand to what extent computational super-resolution can be triggered during deconvolution in order to improve the resolution of the 3D reconstruction of the imaged data

3D deconvolution in Fourier integral microscopy

Fourier integral microscopy (FiMic), also referred to as Fourier light field microscopy (FLFM) in the literature, was recently proposed as an alternative to conventional light field microscopy (LFM). FiMic is designed to overcome the non-uniform lateral resolution limitation specific to LFM. By inserting a micro-lens array at the aperture stop of the microscope objective, the Fourier integral microscope directly captures in a single-shot a series of orthographic views of the scene from different viewpoints. We propose an algorithm for the deconvolution of FiMic data by combining the well known Maximum Likelihood Expectation (MLEM) method with total variation (TV) regularization to cope with noise amplification in conventional Richardson-Lucy deconvolution

Large depth-of-field integral microscopy by use of a liquid Lens

Integral microscopy is a 3D imaging technique that permits the recording of spatial and angular information of microscopic samples. From this information it is possible to calculate a collection of orthographic views with full parallax and to refocus computationally, at will, through the 3D specimen. An important drawback of integral microscopy, especially when dealing with thick samples, is the limited depth of field (DOF) of the perspective views. This imposes a significant limitation on the depth range of computationally refocused images. To overcome this problem, we propose here a new method that is based on the insertion, at the pupil plane of the microscope objective, of an electrically controlled liquid lens (LL) whose optical power can be changed by simply tuning the voltage. This new apparatus has the advantage of controlling the axial position of the objective focal plane while keeping constant the essential parameters of the integral microscope, that is, the magnification, the numerical aperture and the amount of parallax. Thus, given a 3D sample, the new microscope can provide a stack of integral images with complementary depth ranges. The fusion of the set of refocused images permits to enlarge the reconstruction range, obtaining images in focus over the whole region

Three-dimensional real-time darkfield imaging through Fourier lightfield microscopy

We report a protocol that takes advantage of the Fourier lightfield microscopy concept for providing 3D darkfield images of volumetric samples in a single-shot. This microscope takes advantage of the Fourier lightfield configuration, in which a lens array is placed at the Fourier plane of the microscope objective, providing a direct multiplexing of the spatio-angular information of the sample. Using the proper illumination beam, the system collects the light scattered by the sample while the background light is blocked out. This produces a set of orthographic perspective images with shifted spatial-frequency components that can be recombined to produce a 3D darkfield image. Applying the adequate reconstruction algorithm high-contrast darkfield optical sections are calculated in real time. The presented method is applied for fast volumetric reconstructions of unstained 3D samples